The present invention relates generally to the field of manipulation of 3-D objects on computer displays. More particularly, the present invention relates to techniques for simulating life-like user interaction with 3-D objects displayed in computer simulated environments. In accordance with the invention, the 3-D objects are subjected to conflicting goals including the goal of desired object movement by the user such that a resolution of these conflicting goals during user manipulation and resultant movement of the 3-D object creates the impression that the 3-D object is influenced by conflicting physical forces, such as gravity, object initiated motion, etc.
The techniques used in our invention stem from several different backgrounds. Within the domain of 3-D CAD/CAM (Computer Aided Design, Computer Aided Manufacture), a 3-D workspace containing data representations is presented to the user, with the objective of fashioning more 3-D data in accordance with a set of user commands. One manner of interaction in this medium is via a 2-D cursor, where a variety of operations may be performed upon 3-D data. The 3-D position associated with the cursor is derived in a variety of manners, including: use of a secondary form of input; associating the cursor with some additional piece of geometry in the 3-D workspace; interpreting cursor motion as gestures; and augmenting the cursor with additional visual representations inferring additional positioning control. These commands are carried out in a static environment - one where the user selected data and that to which it holds a direct relationship is altered. While the user is interacting with a subset of the data in these environments, other data is not continuously updated in position or orientation unless as a consequence of user interaction. The commands are also carried out as a direct response to the positioning of the cursor. For example, if the cursor is used to select data and then the cursor is moved to a different location, the position of the data is set via a direct mapping from the 2D cursor coordinate to the 3-D data coordinates.
U.S. Pat. No. 5,359,703 describes a technique for operating a system to produce the perception of control of an object in a 3-D workspace presented on a computer display. A mouse or keyboard is used to input movement changes which are directly applied to the 3-D object via a logarithmic function and/or an acceleration function.
Some CAD research has highlighted the use of constraint based systems, whereby 2-D input may be mapped in a manner to 3-D allowing much easier completion of tasks by the user. Explicit relationships are formulated between objects being manipulated and other elements of their environment.
For example, Bukowski, Richard W. and Soquin, Carlo H., xe2x80x9cObject Associations: A Simple and Practical Approach to Virtual 3D Manipulationxe2x80x9d, University of California at Berkeley, 1995 Symposium on Interactive 3D Graphics, Monterey Calif. USA, 1995, describes a process by which an object may be moved above a table and wherein a constraint is inferred attaching the object to the table by some form of pseudo gravity. In the same manner, objects may be attached to other objects or to walls. The viewpoint may be moved while focused upon a specified point upon an object, to enable easy close-up and distant views of the point. Objects may also be moved by applying force, in the same manner as moving them using a rubber band. A similar technique for producing the perception of a moving viewpoint within a 3-D space on a computer display is disclosed in U.S. Pat. No. 5,276,785.
It should be noted that all these advanced proposals still hold the same goal in mind, that to goal being to facilitate the construction and manipulation of objects in a 3D environment by a user, where the user""s goals are to be attained in the easiest possible manner, with minimal interference from other data and entities present (e.g., other users). The prior art techniques described above are inadequate to represent conflicts between entities concerning the manipulation of objects, or to represent entities aiding each other in picking up and moving objects. Moreover, the techniques disclosed in the prior art noted above do not provide a physical sense of actually holding an object, with feedback as to whether the object is heavy or light, or has other such physical properties.
Additional interpretations of 2-D input to create a 3-D position by the use of additional on-screen information are known from the prior art. For example, U.S. Pat. No. 4,808,979 discloses one such method involving symbols to tether a cursor and an object. As another example, U.S. Pat. No. 5,588,098 discloses a method for manipulating an object by use of a displayed bounding region.
These and many other techniques, such as those described in Foley, James and van Dam, Andies, and Feiner, Steven, and Hughes, John, xe2x80x9cComputer Graphics: Principles and Practice, Second Editionxe2x80x9d, Addison Wesley Publishing Company Inc., 1990, all provide improved object manipulation in a 3-D workspace using 2-D input devices. However, these methods are unsuitable for an entertainment product, where the user""s imagination is immersed in the environment. In most of these contexts, such symbols would not be considered a natural part of the environment, breaking the illusion of immersion.
In contrast with the above-described 3-D CAD/CAM domain of interaction, there is real-time 3-D first person perspective entertainment software (referred to as xe2x80x98first-personxe2x80x99) that involves the interaction of a user represented as a collection of data within a 3-D environment. Examples of such xe2x80x98first personxe2x80x99 entertainment software include the products detailed in Abrash, Michael, xe2x80x9cQuake: a Post-Mortem and a Glimpse into the Futurexe2x80x9d, Id Software, Computer Game Developers Conference Proceedings, 1997, and the implementation described in Kolb, Jason, xe2x80x9cWin32 Game Developer""s Guide with DirectX 3xe2x80x9d, The Waite Group, 1997.
In the known 3-D first person entertainment software games, the state of the 3-D environment is being continuously updated at a rate which provides the illusion of a real life occurrence happening at a realistic rate, hence the term xe2x80x9creal-timexe2x80x9d. It is viewed from a first person perspective, as if the viewpoint represents eyes which are a component of the user""s simulated body. The simulation takes place independently to some degree of user interaction. Data representing objects and states in the 3-D space may react to user actions. However, through computer code instructions, objects may also follow a set of objectives in which the user has no part, or which the user must oppose. Codified rules exist as to what should occur when user data and other data interacts, for example, due to proximity. These rules generally involve the collision and repulsion of data, or setting of a given state in either data set via a direct, or physical model. For example, a switch may be thrown.
Input from a mouse, trackball, or keyboard is applied to specify a change in orientation and position of the user viewpoint. In many first-person games, when some form of selection or manipulation input is generated, changes are made to the environment on the basis of what is closest to the center of the entire viewing area at or around that instant in time. Within or about this region, a piece of data is selected by an algorithm, such as proximity to the line of the user viewpoint orientation. The resulting state change is normally bounded by a small series of discrete state changes, often a simple activated/deactivated change.
Such interaction is well suited to conflict in large environments, wherein the objects to be manipulated are widely separated, so as not to be confused if many are present in the user""s view. However, such interaction is poorly suited to the manipulation of detailed spaces containing closely positioned devices or objects which can be controlled. Hence, this type of environment is avoided in the design of such games.
First-person entertainment software has to date avoided the style of interaction present in CAD/CAM applications, and hence the scope and detailed control of such applications. In addition, interaction is normally limited, in that one grasped item is exclusively manipulated by one entity in the environment at any one instant of time.
Virtual Reality (VR) environments normally involve a 3-D environment, whose state is being continuously updated independently, to some degree, of user interaction. Greater interaction is possible in the environment, such as selection of specific data objects while other environmental states are changing. Selection and manipulation may take place using a xe2x80x98data glovexe2x80x99 or other devices providing three or more dimensions of input. Such devices are not widely deployed amongst personal computers (PCs), and are more expensive than the keyboard and mouse usually provided with PCs.
It is also possible to use a 2-D positioned device to select and move objects. Entertainment based VR software systems normally employ limited selection capabilities often akin to first-person software. Some VR software interaction takes a form similar to that of 3-D CAD/CAM, the difference being that other activities may be simultaneously occurring in the space. However, this more detailed interaction is usually applied toward CAD or demonstrational goals. In the case of 2-D interaction, the selected objects are moved by mapping the 2-D position onto the 3-D position of the object in some manner involving exclusive ownership by one user. Examples of such interaction may be found in: Stiles, Randy, xe2x80x9cAdapting VRML 2.0 for Immersive Usexe2x80x9d, Lockheed Martin Advanced Technology Center, VRML 97 Second Symposium on the Virtual Reality Modeling Language, Monterey Calif. USA, 1997; and Harman, Jed and Wernecke, Josie, xe2x80x9cThe VRML 2.0 Handbook: Building Moving Worlds on the Webxe2x80x9d, Silicon Graphics Inc., Addison Wesley Publishing Company Inc., 1996.
As with CAD/CAM, such methods would be inadequate to represent conflicts between plural entities concerning the manipulation of an object simultaneously held by these entities. Moreover, they do not provide a physical sense of actually holding an object, with feedback as to whether the object is heavy or light, or has other such physical properties.
A further category which predates my work is the entertainment software category of 2-D puzzle solving and adventuring. In the 2-D puzzle solving and adventuring entertainment software medium, the simulated environment is presented as a discrete set of images of a 3-D environment, each from a specific position. An example is disclosed in U.S. Pat. No. 4,857,902 which teaches to use video disk data to supply the set images of the 3-D environment. The techniques used in the 2-D puzzle solving and adventuring category of entertainment software are distinct from the above cases in that the majority of the depiction is precalculated before the software is executed. Within this environment, users may move position in discrete steps. The user may look around and zoom in to closely examine a part of the view from this position, using techniques such as those disclosed in U.S. Pat. No. 4,734,690. The user may also pan a viewport window with respect to a virtual screen stored in memory in accordance with the cursor motion techniques disclosed in U.S. Pat. No. 4,720,703. Users may select objects by use of a 2-D cursor. The objects may then be moved in a 2-D, or sometimes a 3-D constrained fashion. Descriptions of this type of environment are given in Mackay, Peter, xe2x80x9cExploring new worlds with Quick Time less than  less than  VRxe2x80x9d, Media Integration, Computer Game Developers Conference Proceedings, 1996, and Stein III, Robert, xe2x80x9cThe making of 11th Hourxe2x80x9d, Computer Game Developers Conference Proceedings, 1996.
However, in the 2-D puzzle solving and adventuring entertainment software medium, objects do not interact by collision or other physical modeling constraints with a variety of other objects in the environment. Instead, object interactions are normally described in terms of specific unique interactions with other objects, illustrated by pre-drawn artwork. This allows for a wide variety of interactions, with the constraint that the interactions that can occur are those explicitly codified and/or drawn during the construction of the product. Interactions based on physical or other general laws are not present. Objects selectable by the user cannot simultaneously and independently be selected and manipulated by other entities in the environment. This is a factor in a frequent criticism of this genre, stating that such games can be very frustrating. A commonly cited reason for this is that objects can only be applied in the manner intended by the designer, not necessarily in the manner desired by the user.
Real time 3-D third person perspective (referred to as xe2x80x98third-personxe2x80x99) entertainment software is distinguished from first-person entertainment software by the viewpoint being loosely associated with and located some distance from the object which represents the user in the space.
In most such third person games, manipulation of the user is quite complex, as a variety of actions are required to complete the tasks specified in the game. However, each action normally requires an input which bears little physical relation to the action performed. Usually a variety of unique button presses or combinations of buttons pressed are required for each such action. This makes the game difficult to master since it requires extensive knowledge of the combinations involved.
Manipulation of objects other than the user is possible in such environments, but is usually fairly simple in nature. Typically, the object representing the presence of the user within the computer simulated environment, referred to as the user object, is first aimed at or in the proximity of the object to be manipulated, followed by one or more button presses or distinct combinations of buttons pressed. As with first-person games, the resulting state change is normally a small series of discrete state changes, often a simple activated/deactivated change. Objects may also be pushed, by moving the user object against the target object and possibly engaging some buttons, in order to move the object in the direction in which the user is moving. Again, as with first-person games, such interaction is well suited to large environments in which objects to be manipulated are widely separated. However, it is poorly suited to fine control, or to easily differentiating between closely spaced objects. For example, it would be difficult or impossible to move only one small object closely ringed by other small objects. As with other entertainment genres, interaction is normally limited, in that one grasped item is exclusively manipulated by one entity in the environment at any one instant of time.
Some third-person games use 2-D cursor input to set the state of data in the environment, but in a direct manner where the effect of a button click is as rapidly as possible translated into a change in a component of the data to which the cursor is pointed. This 2-D input is not a change being made by physically associating the user object with the change or by moving the selected object as if picked up by the user object. Rather, the change is being made by the user acting as some agent external to the user object. This is not applied to the action of objects being grasped by an entity in the environment, nor to multiple entities simultaneously grasping and vying for control of an object.
Accordingly, a general objective of the present invention is to provide techniques for producing the perception of actual manipulation of an object on a display with feedback to indicate whether the object is heavy or light, or possesses other such physical properties.
It is another objective of the present invention to provide techniques for representing and resolving conflicts between two or more entities (users) concerning the manipulation of an object simultaneously held by these entities.
The foregoing and other objectives are provided by a method for user interaction with animated objects displayed in a repeatedly updated 3-D computer simulated environment using 2-D cursor and button input. The present invention allows user manipulation of 3-D objects in a simulation where conflicting goals may be simultaneously applied to the objects and resolved. All goals acting on an object are translated to equivalent information in terms of physical forces, and are applied as forces via a physically based model, where this model has no knowledge as to the source of each force.
Each user is treated as an entity in the environment, with a positional presence. The positional presence of a user is represented by a user object displayed in the simulation. In a preferred embodiment, the user object may take the form of an animated virtual hand. While manipulation on an object is taking place, or at other times, other entities within the simulation may also manipulate the same or other object in the simulation. In other words, more than one entity may manipulate the same data simultaneously. Other forces and entities may be acting upon the representation of the user in the simulation, and elements the user seeks to control. The invention provides a unique interface to a real-time simulation that is easy to use, by virtue of mapping a subset of the inputs to actions consistent with the expectation of the user actually being physically present within the environment. Moreover, the invention can be implemented on a wide range of low cost computing systems, as the requirements are for a minimum of 2-D input and output. The virtual hand can grab, lift, drag, or click on objects within the environment, and be used to allow the user object to climb or otherwise move, in a novel fashion. Thus, other objects representing other users or computer simulated entities can simultaneously and independently manipulate and affect those same or other objects. If an object is not manipulated, the cursor may be used to specify a direction in which the user object should rotate or alter it""s special position, allowing other means of travel through the environment.
In accordance with an advantageous feature of the present invention, other entities within the environment may simultaneously apply forces to an object that is the subject of object selection and drag operations by a particular user. Due to this feature, the other entities may intervene to partially aid or thwart a particular user""s intended goals. Hence, physical conflict for ownership of objects or physical help in performing tasks may be portrayed in a new manner.
Upon selecting an object under defined circumstances, the user may move their own position, as if by pulling or pushing themselves with respect to the selected object, by moving the cursor. This allows the expression of a variety of tasks, such as climbing, in a manner which is easy to use. In addition, control via cursor and weighted force allows a fine degree of control of the user""s position, compared with the button input previously associated with related actions.
Visual feedback as to the weight of a selected object is provided by the degree to which it keeps up with the cursor position, providing this information in an intuitive manner, compared with non-physically based manipulation of a selected object. Given that a selected object has momentum, is light enough, and if it is dragged rapidly and released during this action, the object may also be xe2x80x9cthrownxe2x80x9d into the environment.
Methods and apparatus which incorporate the features described above and which are effective to function as described above constitute specific objectives of this invention.
Other and further objectives of the present invention will be apparent from the following description and claims and are illustrated in the accompanying drawings, which by way of illustration, show preferred embodiments of the present invention and the principles thereof and what are now considered to be the best modes contemplated for applying these principles. Other embodiments of the invention embodying the same or equivalent principles may be used and structural changes may be made as desired by those skilled in the art without departing from the present invention and the purview of the appended claims.